Multivariate Gradient Analysis for Evaluating and Visualizing a Learning System Platform for Computer Programming

This paper explores the application of canonical gradient analysis to evaluate and visualize student performance and acceptance of a learning system platform. The subject of evaluation is a first year BSc module for computer programming. This uses 'Ceebot', an animated and immersive game-like development environment. Multivariate ordination approaches are widely used in ecology to explore species distribution along environmental gradients. Environmental factors are represented here by three 'assessment' gradients; one for the overall module mark and two independent tests of programming knowledge and skill. Response data included Likert expressions for behavioral, acceptance and opinion traits. Behavioral characteristics (such as attendance, collaboration and independent study) were regarded to be indicative of learning activity. Acceptance and opinion factors (such as perceived enjoyment and effectiveness of Ceebot) were treated as expressions of motivation to engage with the learning environment. Ordination diagrams and summary statistics for canonical analyses suggested that logbook grades (the basis for module assessment) and code understanding were weakly correlated. Thus strong module performance was not a reliable predictor of programming ability. The three assessment indices were correlated with behaviors of independent study and peer collaboration, but were only weakly associated with attendance. Results were useful for informing teaching practice and suggested: (1) realigning assessments to more fully capture code-level skills (important in the workplace); (2) re-evaluating attendance-based elements of module design; and (3) the overall merit of multivariate canonical gradient approaches for evaluating and visualizing the effectiveness of a learning system platform.